fix(agentx): preserve trace timing and joins - #31
Open
cquil11 wants to merge 9 commits into
Open
Conversation
Try out this PRQuick install: pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@abf55f902cde0a3e8389c452a83ecbf5ba15dea9Recommended with virtual environment (using uv): uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@abf55f902cde0a3e8389c452a83ecbf5ba15dea9Last updated for commit: |
|
Fern Docs Preview: generation failed — see the Actions log for details. This does not block merge; ask a maintainer to retry if needed. |
cquil11
marked this pull request as ready for review
July 30, 2026 16:49
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
cquil11
force-pushed
the
cquil11/allow-agentx-trace-idle-cap
branch
from
July 31, 2026 16:47
1c315af to
deb5421
Compare
…ynamo#1228) When cache-bust owns trajectory prefix isolation, preserve the same prefix across warmup and profiling so warmup primes the cache used by profiling. Upstream: 9b5f5f7 中文:启用 cache-bust 时,由其负责轨迹前缀隔离,并在预热与性能测试阶段保持相同前缀,使预热能够填充性能测试实际复用的缓存。 Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com> Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Keep the default profiling grace period unless the user explicitly overrides it. Upstream: 5ad0816 中文:除非用户显式覆盖,否则保留默认的性能测试宽限期。 Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com> Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Carry each stream's full recorded delay across the warmup barrier, then subtract one phase-wide minimum so the earliest profiling request starts immediately without changing relative timing, order, or join gates. 中文:在 warmup 屏障后保留每个流完整的记录延迟,再统一减去 profiling 阶段的全局最小值,使最早请求立即启动,同时保持相对时序、请求顺序和 join 门控不变。 Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Keep one replay origin per active trajectory tree during accelerated warmup. Rebuild every surviving root, child, sidecar, and gated join on that shared dataset clock so non-burst profiling preserves cross-stream order and relative spacing. Retain the existing per-stream delay fallback for timestamp-less datasets and seed recycled warmup roots with a fresh origin. Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Register the global drain observer when its phase becomes active so warmup teardown cannot clear profiling ownership. When a per-root cap advances a dependency-blocked timer, retain the expired idle budget and immediately try the next timer without bypassing replay barriers. Signed-off-by: Cam Quilici <cjquilici@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--trace-idle-gap-cap-secondsonly as an observed runtime idle watchdog per complete trajectory treeRuntime semantics
The trace watchdog starts for initial future profiling work and restarts whenever the final in-flight request across a root and all descendants completes. If no request from that tree reaches the wire before the configured cap, AIPerf uniformly advances only that tree's pending scheduler timers. If the first advanced request is still dependency-blocked, the watchdog continues within the same expired idle interval until an eligible request reaches the wire. Other roots are untouched, relative timer spacing is preserved, and dependency gates remain authoritative.
Accelerated warmup intentionally removes runtime delays. At handoff, timestamped roots, subagents, sidecars, and gated joins are restored to their next position on one shared tree clock. Profiling subtracts one global minimum, so the earliest request starts immediately while all remaining streams retain their original flattened order and relative start spacing. Timestamp-less datasets keep the existing per-stream delay fallback.
The global idle observer is registered when each phase becomes active, so warmup teardown cannot clear profiling's observer. The dataset loader does not rewrite timestamps for the trace cap: recorded timestamps,
api_time, and derived end-to-start delays remain unchanged. Actual server completion behavior determines when a runtime tree becomes idle.Validation